Augmented surgical reality environment

ABSTRACT

The present disclosure is directed to an augmented reality surgical system for viewing an augmented image of a region of interest during a surgical procedure. The system includes an image capture device that captures an image of the region of interest. A controller receives the image and applies at least one image processing filter to the image. The image processing filter includes a spatial decomposition filter that decomposes the image into spatial frequency bands. A temporal filter is applied to the spatial frequency bands to generate temporally filtered bands. An adder adds each band spatial frequency band to a corresponding temporally filtered band to generate augmented bands. A reconstruction filter generates an augmented image by collapsing the augmented bands. A display displays the augmented image to a user.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation Application of U.S. patentapplication Ser. No. 16/210,686, filed Dec. 5, 2018, which is aContinuation Application of U.S. patent application Ser. No. 15/327,058,filed Jul. 20, 2015 (now U.S. Pat. No. 10,152,789), which claims thebenefit of and priority to U.S. National Stage Application filed under35 U.S.C. § 371(a) of International Patent Application Serial No.PCT/US2015/041083, filed Jul. 20, 2015, which claims the benefit of andpriority to U.S. Provisional Patent Application No. 62/028,974, filedJul. 25, 2014, the entire disclosure of each of which is incorporated byreference herein.

BACKGROUND 1. Technical Field

The present disclosure relates to surgical techniques to improvesurgical outcomes for a patient. More specifically, the presentdisclosure is directed to systems and methods for augmenting andenhancing a clinician's field of vision while performing a surgicaltechnique.

2. Background of the Related Art

Minimally invasive surgeries have involved the use of multiple smallincisions to perform a surgical procedure instead of one larger opening.The small incisions have reduced patient discomfort and improvedrecovery times. The small incisions have also limited the visibility ofinternal organs, tissue, and other matter.

Endoscopes have been inserted in one or more of the incisions to make iteasier for clinicians to see internal organs, tissue, and other matterinside the body during surgery. These endoscopes have included a camerawith an optical and/or digital zoom capability that is coupled to adisplay showing the magnified view of organs, tissue, and matter insidethe body as captured by the camera. Existing endoscopes and displays,especially those used in surgical robotic systems, have had a limitedability to identify conditions or objects that are within the field ofview of the camera but are not fully visible within the spectrum shownon the display.

For example, existing minimally invasive and robotic surgical tools,including but not limited to endoscopes and displays, have had alimited, if any, ability to identify tissue perfusion after resection,locate different sized arteries within tissue, measure the effectivenessof vessel sealing, identify diseased or dead tissue from a heatsignature, verify appropriate functioning after a resection, distinguishbetween sensitive areas (such as the ureter) and surrounding matter(such as surrounding blood), and detecting super-small leaks that arenot visible with current tests. In some surgeries these checks wereeither not performed or more invasive and/or time consuming tests wereperformed to check for these and other conditions and objects.

There is a need for identifying a greater range of possible conditionsor objects that are within the field of view of a surgical camera butare not fully visible within the spectrum shown on the display duringsurgery.

SUMMARY

In an aspect of the present disclosure, an augmented reality surgicalsystem for viewing an augmented image of a region of interest during asurgical procedure is provided. The system includes an image capturedevice configured to capture an image of the region of interest. Thesystem also includes a controller configured to receive the image andapply at least one image processing filter to the image to generate anaugmented image. The image processing filter includes a spatialdecomposition filter configured to decompose the image into a pluralityof spatial frequency bands, a temporal filter that is configured to beapplied to the plurality of spatial frequency bands to generate aplurality of temporally filtered bands, an adder configured to add eachband in the plurality of spatial frequency bands to a corresponding bandin the plurality of temporally filtered bands to generate a plurality ofaugmented bands, and a reconstruction filter configured to generate anaugmented image by collapsing the plurality of augmented bands. Theaugmented image is then displayed to a user.

The image capture device may capture a video having a plurality of imageframes and the controller applies the at least one image processingfilter to each image frame of the plurality of image frames.

The temporal filter isolates at least one spatial frequency band fromthe plurality of spatial frequency bands to generate the plurality oftemporally filtered bands. The plurality of temporally filtered bandsare amplified by an amplifier before each band in the plurality ofspatial frequency bands is added to a corresponding band in theplurality of temporally filtered bands to generate a plurality ofaugmented bands.

The image processing filter may use an edge detection algorithmconfigured to highlight one or more edges in the image, wherein the oneor more highlighted edges is added to the augmented image.

The system may include at least one hyper-spectral sensor configured toobtain a plurality of hyper-spectral images. The image processing filteruses a hyper-spectral algorithm to combine the plurality of spectralimages to generate a three dimensional hyper-spectral image cube that isadded to the augmented image.

The system may also include an infrared camera configured capture atleast one image, wherein the image processing filter generates aninfrared image that is added to the augmented image.

In another aspect of the present disclosure, methods for generating anaugmented image are provided. A non-transitory computer readable medium,including but not limited to flash memory, compact discs, and solidstate drives, may store instructions that, when executed by a processingdevice, including but not limited to a controller or central processingunit, cause the processing device to perform one or more functions,including the methods described herein. A method may include capturingat least one image using an image capture device. The at least one imageis decomposed to generate a plurality of spatial frequency bands. Atemporal filter is applied to the plurality of spatial frequency bandsto generate a plurality of temporally filtered bands. Each band in theplurality of spatial frequency bands is added to a corresponding band inthe plurality of temporally filtered bands to generate a plurality ofaugmented bands. The plurality of augmented bands is collapsed togenerate an augmented image which is displayed on a display.

At least one spatial frequency band is isolated from the plurality ofspatial frequency bands. The temporally filtered bands may be amplifiedbefore adding each band in the plurality of spatial frequency bands to acorresponding band in the plurality of temporally filtered bands togenerate a plurality of augmented bands.

An edge detection algorithm may be applied to highlight one or moreedges in the image and the one or more highlighted edges is added to theaugmented image.

A plurality of hyper-spectral images may be obtained. The plurality ofhyper-spectral images are combined to generate a three dimensionalhyper-spectral image cube. The three dimensional hyper-spectral imagecube is added to the augmented image.

An infrared image may be obtained and added to the augmented image.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of the presentdisclosure will become more apparent in light of the following detaileddescription when taken in conjunction with the accompanying drawings inwhich:

FIG. 1 is a block diagram of a system for augmenting a surgicalenvironment;

FIGS. 2A-2D are examples of how the system of FIG. 1 may be implemented;

FIG. 3 is a system block diagram of the controller of FIG. 1;

FIG. 4 is a first block diagram of a system for augmenting an image orvideo;

FIG. 5 is a second block diagram of a system for augmenting an image orvideo;

FIG. 6 is a third block diagram of a system for augmenting an image orvideo;

FIG. 7 is a fourth block diagram of a system for augmenting an image orvideo;

and

FIG. 8 is a system block diagram of a robotic surgical system.

DETAILED DESCRIPTION

Image data captured from a surgical camera during a surgical proceduremay be analyzed to identify additional imperceptible properties ofobjects within the camera field of view that may be invisible or visiblebut difficult to clearly see for people viewing the camera imagedisplayed on a screen. Various image processing technologies may beapplied to this image data to identify different conditions in thepatient. For example, Eulerian image amplification techniques may beused to identify wavelength or “color” changes of light in differentparts of a capture image. These changes may be further analyzed toidentify re-perfusion, arterial flow, and/or vessel types.

Eulerian image amplification may also be used to make motion or movementbetween image frames more visible to a clinician. In some instanceschanges in a measured intensity of predetermined wavelengths of lightbetween different image frames may be presented to a clinician to makethe clinician more aware of the motion of particular objects of interest(such as blood).

Image algebra may be used to identify an optimal location for cuttingtissue or other matter during the surgical procedure. In some instances,image algebra may include edge detection and/or Eulerian imageamplification to identify optimal cutting locations.

Hyper-spectral image analysis may be used to identify subtle changes insmall areas within the range of view that may be invisible or otherwisedifficult for the human eye to discern. These hyper-spectral imageanalysis techniques may be combined with Eulerian image amplification toidentify a specific set of changes in these areas.

Image algebra may be combined with hyper-spectral image analysis toidentify an edge of an object or other mass. Image algebra may includeedge detection and/or may be combined with both hyper-spectral imageanalysis and Eulerian image amplification to identify an edge of a mass.

Infrared light may be used to identify a boundary of diseased, dead,and/or abnormal tissue. A filter may be used to isolate one or moredesired wavelengths in an infrared, near infrared, or other range fromcaptured image data. Eulerian image amplification and/or image algebramay be used to analyze the filtered image data and identify a particulartissue boundary.

One or more of Eulerian image amplification, image algebra,hyper-spectral image analysis, and filtering technologies may beincluded as part of an imaging system. These technologies may enable theimaging system to provide additional information about unapparentconditions and objects within a camera's field of view and enhancesurgical outcomes. This additional information may include, but is notlimited to, identifying tissue perfusion, locating arteries of specificsizes (such as larger arteries), verifying an effectiveness of vesselsealing, identifying a heat signature of abnormal tissue, verifyingdesired object motion (such as a lack of movement in edges of deadtissue or verifying proper flow after resection), distinguishing betweensimilar looking objects (such as between the ureter, inferior mesentericartery, and/or surrounding blood), and detecting small leaks (such asleaks that may occur after an anastomosis).

One or more of these technologies may be included as part of an imagingsystem in a surgical robotic system to provide a clinician withadditional information in real time about unapparent conditions andobjects within an endoscope's field of view. This may enable theclinician to quickly identify, avoid, and/or correct undesirablesituations and conditions during surgery. For example, a clinician maybe able to verify during surgery that vessels have been properly sealed,that blood is properly flowing, that there are no air leaks after ananastomosis, and/or that diseased tissue has been removed. The clinicianmay then be able to correct these issues if needed during the surgery. Aclinician may also be able to identify delicate or critical objects inthe body that the surgical instruments should avoid contacting or handleextra carefully, such as larger arteries or the ureter.

The present disclosure is directed to systems and methods for providingan augmented image in real time to a clinician during a surgicalprocedure. The systems and methods described herein apply imageprocessing filters to a captured image to provide an augmented orenhanced image to a clinician via a display. In some embodiments, thesystems and methods permit video capture during a surgical procedure.The captured video is processed in real time or near real time and thendisplayed to the clinician as an augmented image. The image processingfilters are applied to each frame of the captured video. Providing theaugmented image or video to the clinician permits the clinician toidentify and address potential adverse physiologic conditions therebyreducing the need for additional surgical procedures as well as ensuringthe effectiveness of the original surgical procedure.

The embodiments described herein enable a clinician to identify areasreceiving excessive or ineffective blood, effectiveness of stapling orsealing, temperature variations in organs to identify diseased tissue,subtle tissue movement to determine if tissue is alive, and tissuethickness. Additionally, the embodiments described herein may be used toidentify tissue perfusion after resection, location of arteries,distinguish between different tissues, and determine air leaks.

Turning to FIG. 1, a system for augmenting a surgical environment,according to embodiments of the present disclosure, is shown generallyas 100. System 100 includes a controller 102 that has a processor 104and a memory 106. The system 100 also includes an image capture device108, e.g., a camera, that records still frame images or moving images. Asensor array 110 provides information concerning the surgicalenvironment to the controller 102. For instance, sensor array 110includes biometric sensors capable of obtaining biometric data of apatient such as, pulse, temperature, blood pressure, blood oxygenlevels, heart rhythm, etc. Sensor array 110 may also includehyper-spectral sensors to perform hyper-spectral imaging. A display 112,displays augmented images to a clinician during a surgical procedure. Insome embodiments, the controller 102 may communicate with a centralserver (not shown) via a wireless or wired connection. The centralserver may store images of a patient or multiple patients that may beobtained using x-ray, a computed tomography scan, or magnetic resonanceimaging.

FIGS. 2A-2D depict examples of how the system of FIG. 1 is implementedin a surgical environment. As shown in FIGS. 2A-2D, an image capturedevice 108 captures images of a surgical environment during a surgicalprocedure. Images recorded by the image capture device 108, data fromthe sensor array 110, and images from central server (not shown) arecombined by the controller 102 to generate an augmented image that isprovided to a clinician via display 112. As shown in FIGS. 2A-2D,display 112 may be a projector (FIG. 2A), a laser projection system(FIG. 2B), a pair of glasses that projects an image onto one of thelenses such as GOOGLE GLASS® (provided by Google®) (FIG. 2C), bothlenses, or on a facial shield, or a monitor (FIG. 2D). When using amonitor as shown in FIG. 2D, the augmented image is overlaid on an imageof the patient obtained by the image capture device 108.

FIG. 3 depicts a system block diagram of the controller 102. As shown inFIG. 3, the controller 102 includes a transceiver 114 configured toreceive still frame images or video from the image capture device 108 ordata from sensor array 110. In some embodiments, the transceiver 114 mayinclude an antenna to receive the still frame images, video, or data viaa wireless communication protocol. The still frame images, video, ordata are provided to the processor 104. The processor 104 includes animage processing filter 116 that processes the received still frameimages, video, or data to generate an augmented image or video. Theimage processing filter 116 may be implemented using discretecomponents, software, or a combination thereof. The augmented image orvideo is provided to the display 112.

Turning to FIG. 4, a system block diagram of an image processing filterthat may be applied to video received by transceiver 114 is shown as116A. In the image processing filter 116A, each frame of a receivedvideo is decomposed into different spatial frequency bands S₁ to S_(N)using a spatial decomposition filter 118. The spatial decompositionfilter 118 uses an image processing technique known as a pyramid inwhich an image is subjected to repeated smoothing and subsampling.

After the frame is subjected to the spatial decomposition filter 118, atemporal filter 120 is applied to all the spatial frequency bands S₁ toS_(N) to generate temporally filtered bands ST₁ to ST_(N). The temporalfilter 120 is a bandpass filter that is used to extract one or moredesired frequency bands. For example, if the clinician knows thepatient's pulse, the clinician can set the bandpass frequency of thetemporal filter 120, using a user interface (not shown), to magnify thespatial frequency band that corresponds to the patient's pulse. In otherwords, the bandpass filter is set to a narrow range that includes thepatient's pulse and applied to all the spatial frequency bands S₁ toS_(N). Only the spatial frequency band that corresponds to the set rangeof the bandpass filter will be isolated or passed through. All of thetemporally filtered bands ST₁ to ST_(N) are individually amplified by anamplifier having a gain α. Because the temporal filter isolates orpasses through a desired spatial frequency band, only the desiredspatial frequency band gets amplified. The amplified temporally filteredbands ST₁ to ST_(N) are then added to the original spatial frequencybands S₁ to S_(N) to generate augmented bands S′₁ to S′_(N). Each frameof the video is then reconstructed using a reconstruction filter 122 bycollapsing augmented bands S′₁ to S′_(N) to generate an augmented frame.All the augmented frames are combined to produce the augmented video.The augmented video that is shown to the clinician includes a portionthat is magnified, i.e., the portion that corresponds the desiredspatial frequency band, to enable the clinician to easily identify suchportion.

In some embodiments, instead of using an amplifier to amplify theisolated temporally filtered band, the image processing filter 116A mayhighlight the temporally filtered band using a one or more colors beforereconstructing the video. Using a different color for a desired portionof the patient, e.g., a vessel or nerve, may make it easier for theclinician to identify the location of such portion.

Turning to FIG. 5, a system block diagram of an image processing filterthat may be applied to a still frame image or video received bytransceiver 114 is shown as 116B. As shown in FIG. 5, an input image 124(i.e., a captured image or a frame from a video) is inputted into imageprocessing filter 116B. The image processing filter 116B then employs anedge detection algorithm on the inputted image 124 and outputs afiltered image 126 that highlights the edges found in the input image124.

FIG. 6 depicts a block diagram of a system for generating ahyper-spectral image. As shown in FIG. 6, sensor array 110 includeshyper-spectral sensors 128. The hyper-spectral sensors 128 collect a setof images where each image represents a different range of theelectromagnetic spectrum. The set of images are sent to image processingfilter 116C which employs an hyper-spectral algorithm to combine the setof images to form a three-dimensional (3D) hyper-spectral image cube.The 3D hyper-spectral image cube is outputted to the display 112

FIG. 7 depicts a block diagram of a system for generating an infraredimage. As shown in FIG. 7, an infrared camera 130 captures images orvideo and transmits the captured images or video to image processingfilter 116D. Image processing filter 116D processes the receivedcaptured images or video to generate an infrared image that is displayedon display 112.

The image processing filters described above, i.e., 116A-116D, may beused individually to identify physical conditions during a surgicalprocedure. In some embodiments, image processing filter 116A may be usedto identify changes in color in order to identify tissue perfusion orre-perfusion after a resection, arterial flow, vessel types. Imageprocessing filter 116A may also be used to enhance visibility of motionidentify edges of necrotic tissue or appropriate functioning of tissueafter resection.

In some embodiments, the above-described filters may be combined toassist the clinician in identifying adverse physical conditions. Forinstance, image processing filters 116A and 116B may be combined toidentify edges of different tissues to determine the most effectiveplacement for performing a task, e.g., cutting. Image processing filters116A and 116C may be combined to identify subtle changes in small areas,e.g., air leaks that cannot be determined by conventional methods. Imageprocessing filters 116A, 116B, and 116C may be combined to identifyedges of a mass, e.g., a tumor. Image processing filters 116A, 116B, and116D may be combined to identify the boundary of diseased tissue.

Image processing filters 116A, 116, B, 116C, and 116D may be implementedusing different circuits or they may be implemented using a singleprocessor that executes different subroutines based on the filter thatis applied to the image.

The above-described embodiments may also be integrated into a roboticsurgical system. FIG. 8 shows various components that may be included ina robotic surgical system 1, such as two or more robot arms 2, 3; acontrol device 4; and an operating console 5 coupled with control device4. Operating console 5 may include a display device 6, which may be setup in particular to display three-dimensional images; and one or moremanual input devices 7, 8, by means of which a person (not shown), forexample a surgeon, is able to telemanipulate robot arms 2, 3 in a firstoperating mode.

The movement of input devices 7, 8 may be scaled so that a surgicalinstrument attached to a robot arm 2, 3 has a corresponding movementthat is different (e.g. smaller or larger) than the movement of theinput devices 7, 8. The scale factor or gearing ratio may be adjustableso that the clinician can control the resolution of the working ends ofthe surgical instrument.

Each of the robot arms 2, 3 may include a plurality of members, whichare connected through joints, and a surgical assembly 20 to which may beattached, for example, a surgical instrument, such as, for example, animage capture device 108, such as an endoscope, or other surgicalinstrument having an end effector 200, in accordance with any of theembodiments disclosed herein. A distal end of surgical assembly 20 maybe configured to support an image capture device 108 and/or othersurgical instruments having end effectors 200, including, but notlimited to a grasper, surgical stapler, a surgical cutter, a surgicalstapler-cutter, a linear surgical stapler, a linear surgicalstapler-cutter, a circular surgical stapler, a circular surgicalstapler-cutter, a surgical clip applier, a surgical clip ligator, asurgical clamping device, a vessel expanding device, a lumen expandingdevice, a scalpel, a fluid delivery device or any other type of surgicalinstrument. Each of these surgical instruments may be configured foractuation and manipulation by the robot arms 2, 3 via force transmittingmembers. Force transmitting members may be variously configured, suchas, for example, hypotubes, push rods, shafts, or tethers, and cantransmit various forces, such as, for example, axial (i.e., pushing andpulling), rotary, and/or torque. An image capture device 108, such as anendoscope having a camera as an end effector 200 that articulates mayinclude such force transmitting members. One or more of these forcetransmitting members may be configured to control the articulation ofthe camera.

Robot arms 2, 3 may be driven by electric drives that are connected tocontrol device 4. Control device 4 (e.g., a computer) is set up toactivate the drives, in particular by means of a computer program, insuch a way that robot arms 2, 3, their surgical assemblies 20 and thusthe end effector 200 execute a desired movement according to a movementdefined by means of manual input devices 7, 8. Control device 4 may alsobe set up in such a way that it regulates the movement of robot arms 2,3 and/or of the drives.

Control device 4 may also be communicatively coupled to other componentsof the surgical system 1, including, but not limited to, the surgicalassemblies 20; display 6; input devices 7, 8; and surgical instrumentscoupled to robot arms 2, 3 such as image capture device 108 andinstrument having an end effector 200. Control device 4 may also includeor be coupled to controller 102. Controller 102 and/or control device 4may include transceiver 114, which may be configured to receive stillframe images or video from the image capture device 108 or data fromsensor array 110. In some embodiments, the transceiver 114 may includean antenna to receive the still frame images, video, or data via awireless communication protocol. The transceiver 114 may also receivethe still frame images, video, or data via a wired connection. The stillframe images, video, or data may be sent to processor 104. The processor104 includes an image processing filter 116 that processes the receivedstill frame images, video, or data to generate an augmented image orvideo. Processor 104 may include a buffer or memory 106 to store theimages, video, or data being processed. The image processing filter 116may be implemented using discrete components, software, or a combinationthereof. The augmented image or video may be stored and/or sent to thedisplay 6 or another output device.

In the surgical system 1, at least one image capture device 108 may becoupled to at least a first of the two or more robot arms 2, 3. Theimage capture device 108 may be configured to be inserted into thepatient and capture an image of a region of interest inside the patientduring a surgical procedure. The captured image may be displayed on thedisplay 6.

Another surgical instrument having an end effector 200 configured tomanipulate tissue in the region of interest during the surgicalprocedure may be coupled to at least a second of the two or more robotarms 2, 3.

The controller 102 may be configured to process image captured fromimage device 108 and apply at least one image processing filter 116(e.g. filters 116A-116D in one or more of the different ways mentionedthroughout the application) to the captured image to identify animperceptible property of an object in the region of interest during thesurgical procedure. The controller 102 may output the identifiedimperceptible property during the surgical procedure to the clinician.The imperceptible property may be outputted in different ways includingto the display 6 where the imperceptible property may be shown to aclinician and/or by way of haptics so the clinician may feel theimperceptible property. When the imperceptible property is outputted tothe display, the imperceptible property may be altered or transformedinto a more clearly visible signal that may be overlayed onto acorresponding section of the captured image and shown to the clinicianon the display.

Each of the instruments that may be attached to a robot arm 2, 3 may beequipped with a tool-type identifier, such as a quick response code, anidentifier stored in a memory of the instrument, a particular circuitconfiguration associated with the tool-type, and so on. The surgicalsystem 1 may include components or circuitry configured to receiveand/or read the tool-type identifier from each instrument attached to arobot arm 2, 3. This information may then be used to select the specificimage processing filters 116 that may be applied to the captured image.

The tool-type identifier information may be used to identify a surgicalinstrument attached to a robot arm 2, 3 that is located with the fieldof view of the image capture device 108. For example, if a quickresponse code or other tool-type identifier is located on the endeffector 200, shaft, or other component of the surgical instrument thatappears within the field of view of the image device 108, then thecaptured image data may analyzed to identify the surgical instrumentfrom the quick response code or other identifier identified from theimage capture data.

In other instances, a surgical instrument that enters the field of viewof the image capture device 108 may be identified based on a comparisonof positional information about the image capture device 108 (and/or therobot arm 2 to which the image device 108 is attached) and an instrumentattached to one of the other robot arms 3 (and/or the robot arm 3 towhich the instrument is attached). Positional information may beobtained from one or more position sensors in each instrument or in therobot arms 2, 3. A transformation may be used to convert absolutepositional information in different coordinate systems from differentrobot arms 2, 3 so that a relative position of the image capture device108 on one robot arm 2 relative to the surgical instrument attached toanother robot arm 3 may be obtained. The relative position informationmay be used to identify whether the surgical instrument attached to theother robot arm 3 is within the field of view of the image capturedevice 108.

In other instances, one or more cameras may be used to capture an imageof one or more of the robot arms 2, 3. The image data from these camerasmay be analyzed to identify the position of each of the robot arms 2, 3.This positional information may be analyzed to determine if the robotarm 3 to which the surgical instrument is attached is located within thefield of view of the image capture device 108. Other systems and methodsfor determining whether a robot arm 3 to which a surgical instrument isattached is located within the field of view of the image capture device108 may also be used.

If the positional information and/or the tool-type identifier indicatethat a particular surgical instrument is within the field of view of theimage capture device 108, then one or more of the image processingfilters 116 may be automatically selected that correspond to theparticular surgical instrument. For example, if an electro-cauterizationsurgical instrument is identified as being within the field of view ofimage capture device 108, then an image processing filter 116 showing aneffectiveness of a vessel seal may be automatically selected. If theelectro-cauterization instrument is then moved out of the field of viewand a cutting instrument, such as a scalpel in moved into the field ofview, then a different image processing filter 116 showing the locationof large arteries may be automatically selected instead.

Different image processing filters 116 may be automatically selecteddepending on the task that is to be performed. For example, if a cuttingtool is in the field of view and is being moved at a rate exceeding apredetermined threshold and/or a scaling factor of the input device 7,8, is changed so that the surgical instrument moves faster, an imageprocessing filters 116 showing the location of large arteries may beautomatically selected. However, if the cutting tool is being moved aslower rate and/or activated to methodically cut and/or remove tissue,then an image processing filter 116 showing abnormal tissue may be usedinstead. The same analysis may applied to electro-cauterization tools—ifthe tool has not been activated within a predetermined period and/or isbeing moved at a rate exceeding a predetermined threshold then an imageprocessing filter 116 showing the location of large arteries may beautomatically selected. However, if the electro-cauterization tool isbeing moved a slower rate and/or activated within the predeterminedperiod to methodically cut and/or remove tissue, then an imageprocessing filter 116 showing an effectiveness of a vessel seal or otherdesired property may be used instead.

The input device 7, 8 may include haptics 216 to provide feedback to theclinician relating to the imperceptible property. For example, an outputsignal representative of a tissue parameter or condition, e.g., tissueresistance due to manipulation, cutting or otherwise treating, pressureby the instrument onto the tissue, tissue temperature, tissue impedance,and so on, may be generated and transmitted to the input device 7, 8 toprovide haptic feedback to the clinician that varies based on theimperceptible property. Haptics 216 may provide the clinician withenhanced tactile feedback about imperceptible properties of objects thatmay improve patient safety. For example, haptics 216 may be implementedto provide feedback to the clinician when a surgical instrument moved bythe input device 7, 8 comes within a predetermined distance of a largeartery or other delicate tissue to prevent possible injury to the arteryand/or delicate tissue. Haptics 216 may include vibratory motors,electroactive polymers, piezoelectric devices, electrostatic devices,subsonic audio wave surface actuation devices, reverse-electrovibration,or any other device capable of providing a tactile feedback to a user.The input device 7, 8 may also include a variety of different actuatorsfor delicate tissue manipulation or treatment further enhancing theclinician's ability to mimic actual operating conditions.

The embodiments disclosed herein are examples of the disclosure and maybe embodied in various forms. Specific structural and functional detailsdisclosed herein are not to be interpreted as limiting, but as a basisfor the claims and as a representative basis for teaching one skilled inthe art to variously employ the present disclosure in virtually anyappropriately detailed structure. Like reference numerals may refer tosimilar or identical elements throughout the description of the figures.

The phrases “in an embodiment,” “in embodiments,” “in some embodiments,”or “in other embodiments,” which may each refer to one or more of thesame or different embodiments in accordance with the present disclosure.A phrase in the form “A or B” means “(A), (B), or (A and B)”. A phrasein the form “at least one of A, B, or C” means “(A), (B), (C), (A andB), (A and C), (B and C), or (A, B and C)”. A clinician may refers to aclinician or any medical professional, such as a doctor, nurse,technician, medical assistant, or the like) performing a medicalprocedure.

The systems described herein may also utilize one or more controllers toreceive various information and transform the received information togenerate an output. The controller may include any type of computingdevice, computational circuit, or any type of processor or processingcircuit capable of executing a series of instructions that are stored ina memory. The controller may include multiple processors and/ormulticore central processing units (CPUs) and may include any type ofprocessor, such as a microprocessor, digital signal processor,microcontroller, or the like. The controller may also include a memoryto store data and/or algorithms to perform a series of instructions.

Any of the herein described methods, programs, algorithms or codes maybe converted to, or expressed in, a programming language or computerprogram. A “Programming Language” and “Computer Program” includes anylanguage used to specify instructions to a computer, and includes (butis not limited to) these languages and their derivatives: Assembler,Basic, Batch files, BCPL, C, C+, C++, Delphi, Fortran, Java, JavaScript,Machine code, operating system command languages, Pascal, Perl, PL1,scripting languages, Visual Basic, metalanguages which themselvesspecify programs, and all first, second, third, fourth, and fifthgeneration computer languages. Also included are database and other dataschemas, and any other meta-languages. No distinction is made betweenlanguages which are interpreted, compiled, or use both compiled andinterpreted approaches. No distinction is also made between compiled andsource versions of a program. Thus, reference to a program, where theprogramming language could exist in more than one state (such as source,compiled, object, or linked) is a reference to any and all such states.Reference to a program may encompass the actual instructions and/or theintent of those instructions.

Any of the herein described methods, programs, algorithms or codes maybe contained on one or more machine-readable media or memory. The term“memory” may include a mechanism that provides (e.g., stores and/ortransmits) information in a form readable by a machine such a processor,computer, or a digital processing device. For example, a memory mayinclude a read only memory (ROM), random access memory (RAM), magneticdisk storage media, optical storage media, flash memory devices, or anyother volatile or non-volatile memory storage device. Code orinstructions contained thereon can be represented by carrier wavesignals, infrared signals, digital signals, and by other like signals.

It should be understood that the foregoing description is onlyillustrative of the present disclosure. Various alternatives andmodifications can be devised by those skilled in the art withoutdeparting from the disclosure. For instance, any of the augmented imagesdescribed herein can be combined into a single augmented image to bedisplayed to a clinician. Accordingly, the present disclosure isintended to embrace all such alternatives, modifications and variances.The embodiments described with reference to the attached drawing figs.are presented only to demonstrate certain examples of the disclosure.Other elements, steps, methods and techniques that are insubstantiallydifferent from those described above and/or in the appended claims arealso intended to be within the scope of the disclosure.

What is claimed is:
 1. An image processing filter of an augmentedreality surgical system for viewing an augmented image of a region ofinterest, the image processing filter comprising: a spatialdecomposition filter configured to decompose an image of a region ofinterest inside a patient into a plurality of spatial frequency bands; atemporal filter configured to be applied to the plurality of spatialfrequency bands to generate a plurality of temporally filtered bands; anadder configured to add each band in the plurality of spatial frequencybands to a corresponding band in the plurality of temporally filteredbands to generate a plurality of augmented bands; and a reconstructionfilter configured to generate the augmented image by collapsing theplurality of augmented bands.
 2. The image processing filter of claim 1,wherein the image processing filter applies the at least one imageprocessing filter to each image frame of a plurality of image frames ofthe image of a region of interest.
 3. The image processing filter ofclaim 1, wherein the temporal filter includes a bandpass filter.
 4. Theimage processing filter of claim 3, wherein a bandpass frequency of thebandpass filter is set by a clinician.
 5. The image processing filter ofclaim 1, wherein the temporal filter isolates at least one spatialfrequency band from the plurality of spatial frequency bands to generatethe plurality of temporally filtered bands.
 6. The image processingfilter of claim 1, wherein the plurality of temporally filtered bands isamplified by an amplifier before each band in the plurality of spatialfrequency bands is added to the corresponding band in the plurality oftemporally filtered bands to generate the plurality of augmented bands.7. The image processing filter of claim 1, wherein the image processingfilter uses an edge detection algorithm configured to highlight one ormore edges in the image, and the one or more highlighted edges is addedto the augmented image.
 8. The image processing filter of claim 1,wherein the image processing filter uses a hyper-spectral algorithm tocombine a plurality of spectral images to generate a three dimensionalhyper-spectral image cube that is added to the augmented image.
 9. Theimage processing filter of claim 1, wherein the image processing filtergenerates an infrared image from the image, and the infrared image isadded to the augmented image.
 10. A method for generating an augmentedimage of a region of interest during a surgical procedure, the methodcomprising: decomposing at least one image to generate a plurality ofspatial frequency bands; applying a temporal filter to the plurality ofspatial frequency bands to generate a plurality of temporally filteredbands; adding each band in the plurality of spatial frequency bands to acorresponding band in the plurality of temporally filtered bands togenerate a plurality of augmented bands; collapsing the plurality ofaugmented bands to generate the augmented image; and displaying theaugmented image on a display.
 11. The method of claim 10, furthercomprising isolating at least one spatial frequency band from theplurality of spatial frequency bands.
 12. The method of claim 11,further comprising amplifying the temporally filtered bands beforeadding each band in the plurality of spatial frequency bands to acorresponding band in the plurality of temporally filtered bands togenerate a plurality of augmented bands.
 13. The method of claim 10,further comprising: applying an edge detection algorithm configured tohighlight one or more edges in the image; and adding the one or morehighlighted edges to the augmented image.
 14. The method of claim 10,further comprising: adding a three dimensional hyper-spectral imagecube, generated from a plurality of hyper-spectral images, to theaugmented image.
 15. The method of claim 10, further comprising: addingan infrared image to the augmented image.
 16. A non-transitory computerreadable medium storing instructions that, when executed by a processingdevice, cause the processing device to: decompose an image of a regionof interest inside a patient into a plurality of spatial frequencybands; apply a temporal filter to the plurality of spatial frequencybands to generate a plurality of temporally filtered bands; add eachband in the plurality of spatial frequency bands to a corresponding bandin the plurality of temporally filtered bands to generate a plurality ofaugmented bands; and generate the augmented image by collapsing theplurality of augmented bands.
 17. The non-transitory computer readablemedium of claim 16, further comprising additional stored instructionsthat, when executed by the processing device, cause the processingdevice to isolate at least one spatial frequency band containinginformation about an imperceptible property from the plurality ofspatial frequency bands.
 18. The non-transitory computer readable mediumof claim 17, further comprising additional stored instructions that,when executed by the processing device, cause the processing device toamplify the temporally filtered bands before adding each band in theplurality of spatial frequency bands to a corresponding band in theplurality of temporally filtered bands.
 19. The non-transitory computerreadable medium of claim 17, further comprising additional storedinstructions that, when executed by the processing device, cause theprocessing device to: apply an edge detection algorithm configured tohighlight one or more edges in the image data, the one or more edgesbeing the imperceptible property; and add the one or more highlightededges to the augmented image.
 20. The non-transitory computer readablemedium of claim 16, further comprising additional stored instructionsthat, when executed by the processing device, cause the processingdevice to: combine a plurality of spectral images to generate a threedimensional hyper-spectral image cube that is added to the augmentedimage.